Micro-expression recognition method based on brain-like vision

Abstract

The invention provides a micro-expression recognition method based on brain-like vision. The method comprises the following steps of: dividing an event flow output by an event camera according to fixed time steps, constructing a multi-time-step event frame sequence, and synchronously acquiring an initial grayscale image as static texture prior; extracting high-dimensional features of each time step event frame through an event encoder sharing parameters, and generating a corresponding pixel-level motion vector field; performing deformation sampling on the initial grayscale image by using the motion vector field to generate a reconstructed image of the target time step; and splicing the initial grayscale image and the deformed image in a channel dimension, inputting the spliced image into an image refining network for fusion optimization, and outputting a continuous grayscale image sequence. According to the method, the asynchronous event data output by the event camera can be effectively converted into a high-quality continuous image sequence, and the time sequence modeling precision and robustness of micro-expression recognition are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and brain-like perception, belonging to the interdisciplinary fields of image processing, event vision and emotion computing, and particularly to a micro-expression recognition method based on brain-like vision. Background Technology

[0002] With the rapid development of artificial intelligence and brain-like perception technology, micro-expression recognition, as an important research direction in the field of affective computing, is widely used in scenarios such as intelligent security, human-computer interaction, and clinical psychological assessment.

[0003] In related technologies, a micro-expression recognition system based on image frame sequences has been constructed through the collaborative operation of traditional image sensors and deep learning models. Specifically, this system covers the entire process from image acquisition, preprocessing, feature extraction to temporal modeling and emotion classification, including key modules such as convolutional neural networks (CNN) and recurrent neural networks (RNN). However, existing image frame-driven methods directly adopt a fixed time interval sampling mechanism, which does not fully consider the instantaneous and dynamic nature of micro-expressions. This may lead to problems such as misaligned expression frames and loss of details, or defects such as information redundancy and excessive power consumption in high-dynamic scenes, thus affecting recognition accuracy and system real-time performance. In addition, when processing microsecond-level facial changes, traditional methods are limited by the frame rate and dynamic range of image sensors, making it difficult to meet the robust recognition requirements under complex lighting and occlusion conditions, thus limiting their application potential in edge computing and low-power devices. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a micro-expression recognition method based on brain-like vision. It aims to solve the following technical problems: (1) Overcoming bandwidth and power consumption limitations: Visual data is acquired through an event camera, and the dynamic process of facial micro-expressions is perceived under low power consumption and low redundancy information flow conditions; (2) Completing visual temporal information: A multi-time step image reconstruction module is designed to reconstruct the corresponding single image frame based on event data of multiple time steps, and the single image frame is converted into a continuous grayscale image sequence to effectively reconstruct the process of facial micro-expression changes; (3) High-precision recognition model construction: A temporal spiking neural network model based on image-event fusion is constructed to achieve robust recognition of micro-expressions.

[0006] The second objective of this invention is to propose a micro-expression recognition device based on brain-like vision.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a micro-expression recognition method based on brain-like vision, comprising: S1, divide the event stream output by the event camera into fixed time steps, construct a multi-time step event frame sequence, and synchronously acquire the initial grayscale image as a static texture prior; S2, high-dimensional features of event frames at each time step are extracted by an event encoder with shared parameters to generate corresponding pixel-level motion vector fields; S3, use the motion vector field to deform and sample the initial grayscale image to generate the reconstructed image of the target time step; S4 concatenates the initial grayscale image and the deformed image along the channel dimension, and the input image thinning network performs fusion optimization to output a continuous grayscale image sequence.

[0008] In one embodiment of the present invention, S1 includes: S11, the event stream is segmented according to a preset time step to generate event frames with multiple time steps; S12, align the event frame with the synchronously acquired initial grayscale image in the time dimension to ensure that the timestamp error between the event frame and the image frame is less than a preset time threshold.

[0009] In one embodiment of the present invention, S2 includes: S21, the event encoder consists of multiple convolutional layers, batch normalization layers and nonlinear activation functions, used to extract spatial edge information, local motion cues and event intensity distribution of event frames; S22, the motion vector field output by the motion field estimation network is normalized to the interval [-1, 1] using the tanh activation function to adapt to the subsequent bilinear interpolation operation.

[0010] In one embodiment of the present invention, S4 includes: S41, the image thinning network consists of multiple convolutional layers, normalization layers and nonlinear activation functions, used to remove artifacts in the reconstructed image and enhance texture details; S42, the number of input channels of the image thinning network is the sum of the number of channels of the initial image and the number of channels of the deformed image, and the number of output channels is the same as the number of channels of the initial image.

[0011] In one embodiment of the present invention, it further includes: S5 performs multimodal fusion of the reconstructed image sequence and event frame sequence, and inputs it into a spiking neural network for temporal feature extraction and emotion classification; S6 uses the LIF neuron model for the spiking neural network and trains it using BPTT combined with the surrogate gradient method.

[0012] To achieve the above objectives, a second aspect of the present invention provides a micro-expression recognition device based on brain-like vision, comprising: The event frame construction module is used to divide the event stream output by the event camera into fixed time steps, construct a multi-time step event frame sequence, and synchronously acquire the initial grayscale image as a static texture prior. The event feature extraction module is used to extract high-dimensional features of event frames at each time step through an event encoder with shared parameters, and generate corresponding pixel-level motion vector fields. The image deformation sampling module is used to perform deformation sampling on the initial grayscale image using the motion vector field to generate a reconstructed image at the target time step; The image fusion optimization module is used to stitch the initial grayscale image and the deformed image together in the channel dimension. The input image thinning network performs fusion optimization and outputs a continuous grayscale image sequence.

[0013] The method and apparatus of this invention can convert the asynchronous event stream output by the event camera into a high-quality continuous image sequence, effectively improving the temporal modeling accuracy and robustness of micro-expression recognition, while reducing system power consumption and bandwidth requirements.

[0014] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a micro-expression recognition method based on brain-like vision provided in this application embodiment; Figure 2 A framework diagram of an image reconstruction algorithm based on multi-time-step event data provided in this application embodiment; Figure 3 A framework diagram of a multimodal micro-expression recognition algorithm that fuses event and image data provided in embodiments of this application; Figure 4 This is a functional block diagram of a neuromorphic heterogeneous computing micromodule provided in an embodiment of this application; Figure 5 This is a structural outline diagram of a neuromorphic heterogeneous computing micromodule provided in an embodiment of this application; Figure 6 This is a structural diagram of a micro-expression recognition device based on brain-like vision, provided in an embodiment of this application. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The following description, with reference to the accompanying drawings, describes a method and apparatus for micro-expression recognition based on brain-like vision, according to an embodiment of the present invention.

[0019] Example 1 Figure 1 This is a flowchart of a micro-expression recognition method based on brain-like vision according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1 divides the event stream output by the event camera into fixed time steps, constructs a multi-time-step event frame sequence, and synchronously acquires the initial grayscale image as a static texture prior.

[0020] Specifically, this step aims to divide the event stream output by the event camera into fixed time steps, construct a multi-time-step event frame sequence, and simultaneously acquire an initial grayscale image as a static texture prior, providing structured input for subsequent image reconstruction and micro-expression recognition. In some implementations, the event camera operates in an event-driven manner, with each event containing spatial coordinates (x, y), a timestamp (t), and polarity (ON / OFF), exhibiting microsecond-level temporal resolution and high dynamic range. Since the event stream is asynchronous and non-uniformly distributed in time, it needs to be transformed into a fixed-time-step event frame sequence through a time window partitioning strategy to facilitate processing by deep learning models.

[0021] In practice, the event stream is divided into multiple time steps, each with a length of T=24ms. The choice of this time step length is based on the average duration of micro-expressions (usually less than 500ms) and the sampling characteristics of the event camera, ensuring that sufficient event density is accumulated within each time step to reflect local dynamic changes. The event frames are constructed as follows: within each time window, the event is spatially integrated to generate an event image for the corresponding time step, where the value of each pixel represents the cumulative intensity and direction information of the event within that time step.

[0022] Simultaneously, to provide the static background information required for image reconstruction, the system synchronously acquires an initial grayscale image aligned with the event stream time. This image is typically a high-resolution image frame at the start of the event sequence, used to provide prior information such as static texture and illumination distribution of the face, thereby enhancing the spatial fidelity and structural consistency of the reconstructed image during subsequent image deformation and fusion processes.

[0023] This step plays a crucial role in the entire technical solution. The multi-time-step event frame sequence it constructs, together with the initial grayscale image, forms the input basis for the image reconstruction module. It provides spatiotemporally aligned data support for motion field estimation, deformation sampling, and image fusion, and is the core prerequisite for realizing micro-expression recognition under low power consumption and high dynamic range.

[0024] Furthermore, S1 includes: S11, the event stream is segmented according to a preset time step to generate event frames with multiple time steps.

[0025] Specifically, this step involves segmenting the event stream output by the event camera into 24-microsecond (μs) time steps to generate event frames for multiple time steps. This operation is a crucial step in event data preprocessing in image reconstruction and micro-expression recognition systems, aiming to transform continuous, asynchronously triggered event streams into discrete event frame sequences with temporal structure, thereby adapting them to the input format of subsequent deep learning models.

[0026] In some implementations, the event stream is asynchronously output by the event camera with sub-millisecond temporal precision (typically 1μs to 100μs). Each event contains pixel coordinates (x, y), a timestamp (t), and polarity (ON / OFF). Due to the non-uniform temporal characteristics of the event data, a fixed time step (T=24μs) is introduced for event aggregation. Specifically, a sliding time window is set with a step size of 24μs. The event stream is time-sliced, and all events falling within this time window are statistically analyzed in the spatial dimension to generate an event image for the corresponding time step. The event image is typically represented using two channels, recording the intensity distribution of ON and OFF events separately to reflect the positive and negative directions of brightness changes.

[0027] The choice of a time step T=24μs is based on the typical response latency of the event camera and the timescale of micro-expression dynamic changes (usually within 200ms), ensuring that the temporal complexity of the event data is reduced without losing key dynamic information. The resolution of the event image is usually consistent with the pixel array of the event camera, such as 128×128 or 240×180, and the output dimension of each event frame is (H×W×2), where H and W are the height and width of the image, respectively.

[0028] This step primarily transforms the event stream into a sequence of event frames that can be processed by the image reconstruction network, providing structured input for subsequent motion field estimation, image deformation, and fusion. By generating event frames with a fixed time step, the system can achieve high temporal resolution modeling of micro-expressions under low bandwidth and low power consumption conditions, thereby improving the recognition model's ability to perceive instantaneous facial expression changes.

[0029] This step effectively solves the mismatch between event data and image frames in the time dimension, providing continuous temporal input for image reconstruction algorithms and enhancing the model's accuracy in modeling the dynamic process of micro-expressions. Simultaneously, through a fine time step of 24μs, the system can capture the instantaneous changes in micro-expressions, laying the foundation for subsequent temporal processing by spiking neural networks.

[0030] S12, align the event frame with the synchronously acquired initial grayscale image in the time dimension to ensure that the timestamp error between the event frame and the image frame is less than a preset time threshold.

[0031] Specifically, this step aims to precisely align the event frames output by the event camera with the synchronously acquired initial grayscale image in the time dimension, ensuring that the timestamp error between the two is less than 10 microseconds. This provides highly time-consistent data input for subsequent image reconstruction and micro-expression recognition. In some implementations, this alignment process relies on a hardware-level time synchronization mechanism between the event camera and the image sensor, typically achieved through a shared clock source (such as PCLK in the MIPI-CSI interface or a system-level clock) to uniformly calibrate the timestamps. The event camera responds to changes in ambient brightness in sub-millisecond increments, and each event contains a precise timestamp (typically with an accuracy of 1 microsecond), while image frames are acquired at fixed time intervals (such as 30ms or 60ms). To achieve time alignment between event frames and image frames, the system first sorts the event stream by timestamp and, using the image frames as a reference, divides the event data into multiple time windows (T=24) with time intervals consistent with the image frame intervals. Within each window, the timestamp of the event frame must have an error of less than ±5 microseconds with the corresponding image frame's timestamp to meet the stringent time accuracy requirements of micro-expression recognition.

[0032] Furthermore, this alignment process can be implemented using timestamp interpolation or event resampling algorithms, such as linear interpolation or a timestamp-based sliding window averaging strategy, to eliminate temporal offsets between the event stream and image frames. In practical applications, this step is typically implemented in FPGAs or neuromorphic computing units (such as LYNX KA200) to ensure low latency and high real-time performance. This alignment mechanism is a crucial preliminary step in image reconstruction algorithms, directly affecting the accuracy of motion field estimation and image deformation, thereby improving the robustness and recognition accuracy of micro-expression recognition models in dynamic scenes.

[0033] S2 extracts high-dimensional features of event frames at each time step through an event encoder with shared parameters, and generates corresponding pixel-level motion vector fields.

[0034] Specifically, extracting high-dimensional features of event frames at each time step through a shared-parameter event encoder and generating corresponding pixel-level motion vector fields is one of the key steps in the image reconstruction and micro-expression recognition system of this invention. This step aims to utilize the high temporal resolution event stream output by the event camera to extract feature representations with spatiotemporal semantics, providing precise motion guidance for subsequent image deformation and fusion.

[0035] The event encoder employs a multi-layer convolutional neural network structure, including convolutional layers (Conv2D), batch normalization layers (BatchNorm2D), and non-linear activation functions (such as ReLU or SELU). Its channel count is set between 64 and 256, depending on the spatial resolution of the event frames and the task complexity. Event frames are accumulated by the event camera over a fixed time step (T=24). Each event contains spatial coordinates (x, y), a timestamp (t), and polarity (p), representing the direction of brightness change of the pixel. The event encoder extracts features from the event image at each time step, outputting a high-dimensional feature map with local motion cues, edge information, and event intensity distribution. To ensure temporal consistency, all event frames share the same set of encoder parameters, thus maintaining the consistency and continuity of feature representation in temporal modeling.

[0036] This step is widely used in facial micro-expression recognition systems in low-power edge devices. Leveraging the high dynamic response capability of the event camera, the system can stably extract motion features of micro-expressions in complex environments such as strong light and rapid movement, thereby improving recognition robustness. This motion vector field can be used as input to an image deformation sampling network to reconstruct continuous grayscale image sequences, providing high-quality temporal input for subsequent spiking neural network (SNN) classifiers.

[0037] The technical value of this step lies in the fact that by using an event encoder with shared parameters, the complexity of the model is effectively reduced and the temporal modeling capability is improved. At the same time, the generated pixel-level motion vector field provides a precise spatial transformation basis for image reconstruction, which significantly enhances the spatiotemporal perception accuracy and computational efficiency of the micro-expression recognition system.

[0038] Furthermore, S2 includes: S21, the event encoder consists of multiple convolutional layers, batch normalization layers and nonlinear activation functions, used to extract spatial edge information, local motion cues and event intensity distribution of event frames.

[0039] Specifically, the event encoder, as a key component in the image reconstruction module, has the core function of extracting spatial edge information, local motion cues, and event intensity distribution from event frames, providing high-dimensional, learnable feature representations for subsequent motion field estimation and image deformation. In some implementations, the event encoder consists of multiple convolutional layers, batch normalization (BN) layers, and nonlinear activation functions (such as ReLU or SELU), forming an end-to-end trainable feature extraction network.

[0040] Specifically, event frames are accumulated by the event camera over a fixed time step (T=24). Their input dimension is typically H×W, where 2 represents the ON / OFF event channels, and H×W is the image resolution. The event encoder extracts multi-scale features from the event image by stacking multiple convolutional layers (Conv2D). Each convolutional operation uses a 3×3 kernel with a stride of 1 or 2, and the number of channels increases sequentially, for example, from the initial 2 channels to 64, 128, 256, etc., to enhance feature representation. Batch normalization (BN) layers follow the convolutional layers to accelerate the training process, alleviate the vanishing gradient problem, and improve the model's generalization ability. Non-linear activation functions are used to introduce non-linear mappings, enhancing the network's ability to model complex features.

[0041] The feature map output by the event encoder has a high-dimensional representation with C channels, typically set to 256 or 512, to fully capture edge, motion, and intensity information in the event image. This feature map not only preserves the spatial structure of the event data but also ensures semantic consistency of event features across different time steps through temporal consistency constraints (such as shared encoder parameters), thus providing stable and continuous input for subsequent motion field estimation. This step acts as a bridge in the entire image reconstruction process, and its output directly affects the accuracy of motion field estimation and the quality of image reconstruction. It is a key technical step in achieving low power consumption and high dynamic response in micro-expression recognition systems.

[0042] S22, the motion vector field output by the motion field estimation network is normalized to the interval [-1, 1] using the tanh activation function to adapt to the subsequent bilinear interpolation operation.

[0043] Specifically, in the image deformation and fusion step, the motion vector field output by the motion field estimation network needs to be normalized using the tanh activation function, mapping it to the interval [-1, 1] to accommodate the subsequent bilinear interpolation operation. This step plays a crucial bridging role in the image reconstruction process, ensuring that the motion vectors have a reasonable numerical range during spatial transformation, thereby improving the accuracy and stability of image deformation.

[0044] Motion field estimation networks typically consist of multiple convolutional and deconvolutional layers, with their output being a two-dimensional tensor representing the displacement of each pixel in the horizontal and vertical directions. Due to the high temporal resolution and sparsity of event data output by event cameras, motion field estimation networks need to be highly sensitive to local motion information. Before normalization, the motion vectors output by the network are usually in an arbitrary range of real numbers, potentially exceeding the boundaries of the image coordinate system, leading to interpolation failures or invalid sampling points. Therefore, a tanh activation function is introduced after the output layer.

[0045] The tanh activation function outputs in the range [-1, 1], which matches the normalized coordinate range required by the grid_sample function in PyTorch. This function requires the input sampling grid coordinates to be within the range [-1, 1], where -1 represents the top-left corner of the image and 1 represents the bottom-right corner. Tanh normalization ensures that motion vectors are effectively interpolated within the image boundaries, avoiding sampling failures or image distortion due to exceeding the range.

[0046] This step is widely used in event camera-driven image reconstruction tasks, especially in micro-expression recognition systems. Due to the high dynamic response characteristics of event data, accurate normalization of motion vectors is crucial for reconstructing continuous, high-fidelity image sequences. In low-power edge devices, this normalization operation can effectively reduce computational resource consumption while improving the robustness to image deformation.

[0047] Tanh normalization not only improves the numerical stability of motion field estimation, but also enhances the spatial continuity and temporal consistency of image deformation, providing high-quality input sequences for subsequent image fusion and micro-expression recognition, thereby significantly improving the overall system's recognition accuracy and response efficiency.

[0048] S3, the initial grayscale image is deformed and sampled using the motion vector field to generate a reconstructed image at the target time step.

[0049] Specifically, the step of "using the motion vector field to perform deformation sampling on the initial grayscale image to generate a reconstructed image at the target time step" is a key link in the image reconstruction module. Its core objective is to perform spatial transformation on the initial grayscale image through event-driven motion estimation results, thereby generating a high-quality image frame with temporal consistency and spatial fidelity at the target time step.

[0050] At the technical implementation level, this step first uses the multi-timestep event features extracted by the event encoder to generate a pixel-level motion vector field (MVF) frame by frame through a motion estimation network. This motion vector field represents the displacement information of each pixel from the initial image to the current timestep image, and is typically a two-dimensional tensor with the shape of... ,in , Here, represents the height and width of the image, and 2 represents the horizontal and vertical displacements, respectively. Motion field estimation networks typically consist of multiple convolutional layers and nonlinear activation functions; the output motion vector is then processed... Activation function normalized to The interval is set so that it can be overlaid with the standard sampling grid.

[0051] Subsequently, a standard two-dimensional sampling grid is constructed, with its original coordinates as follows: Expanded through tensor operations to The format in which The batch size is determined by adding the normalized motion vector to the standard mesh to obtain the deformed sampled coordinates, which are then mapped to the desired values ​​using coordinate normalization. The range is adjusted to fit the bilinear interpolation function in deep learning frameworks (such as `grid_sample` in PyTorch). Finally, the deformed sampling grid is bilinearly interpolated with the initial grayscale image to generate the reconstructed image at the target time step.

[0052] The displacement accuracy of the motion vector field is typically controlled at the pixel level, meaning the displacement in each direction does not exceed one pixel. The normalization operation of the sampling grid follows image coordinate normalization standards to ensure the stability and continuity of the interpolation process. Furthermore, the output image resolution of this step is consistent with the initial grayscale image, typically [resolution value missing]. or This is to match the input requirements of subsequent micro-expression recognition models.

[0053] This step primarily involves fusing the sparse event stream output from the event camera with the initial image to generate a continuous grayscale image sequence, thereby providing high-quality temporal input for a micro-expression recognition model based on a spiking neural network (SNN). Especially in low-power, high-dynamic-range edge devices, this method can achieve high-precision reconstruction and recognition of micro-expressions without relying on high-frame-rate image acquisition.

[0054] By employing event-driven motion estimation and deformable sampling mechanisms, this method effectively compensates for the image loss problem in the temporal dimension of event cameras, improving the temporal continuity and spatial fidelity of image sequences and providing a reliable data foundation for subsequent temporal modeling and sentiment classification. Furthermore, this method is end-to-end trainable and can be co-optimized with the overall image reconstruction network, further enhancing the system's robustness under complex lighting and dynamic scenes.

[0055] S4 concatenates the initial grayscale image and the deformed image along the channel dimension, and the input image thinning network performs fusion optimization to output a continuous grayscale image sequence.

[0056] Specifically, in the image thinning stage, the initial grayscale image and the deformed image are stitched together along the channel dimension, and the input image thinning network is fused and optimized to finally output a continuous grayscale image sequence. This step is a key link in the image reconstruction process, aiming to improve the spatial detail and temporal consistency of the reconstructed image through multimodal feature fusion, thereby providing high-quality temporal image input for subsequent micro-expression recognition.

[0057] In some implementations, the initial grayscale image serves as static background information, providing the facial texture structure and prior knowledge, while the deformed image is generated by an event-driven motion field estimation module, reflecting the pixel-level spatial transformation at the target time step. The two are concatenated along the channel dimension to form an input tensor with rich spatiotemporal information.

[0058] The refinement network typically consists of multiple convolutional neural networks (CNNs), including BatchNorm layers and non-linear activation functions (such as ReLU or LeakyReLU) to extract features and enhance details from the stitched image. The reconstructed image output by the network has the same spatial resolution as the input image and maintains continuity in the temporal dimension, meeting the high-precision requirements of micro-expression recognition tasks for temporal information.

[0059] In practical applications, this step is particularly suitable for low-power vision systems that fuse event cameras with traditional image sensors. By introducing motion-field-guided deformed images and joint optimization of the initial image, image blurring or artifacts caused by event-driven sampling can be effectively eliminated, improving the visual fidelity and dynamic expression of image sequences. Technically, this fusion mechanism significantly enhances the structural clarity and temporal continuity of images, providing more stable and richer input features for subsequent micro-expression recognition models based on spiking neural networks (SNNs), thereby improving overall recognition accuracy and robustness.

[0060] Furthermore, S4 includes: S41, the image thinning network, consists of multiple convolutional layers, normalization layers, and nonlinear activation functions, used to remove artifacts in the reconstructed image and enhance texture details.

[0061] Specifically, the image thinning network is a key component of the image reconstruction module of this invention. Its core function is to fuse static background information with dynamically event-driven deformed images to remove artifacts that may be introduced during the reconstruction process and enhance the texture details of the image. This network consists of multiple convolutional layers, normalization layers (such as Batch Normalization), and nonlinear activation functions (such as ReLU or Leaky ReLU), and has end-to-end trainable characteristics, enabling it to adaptively optimize image quality.

[0062] In the implementation, the input to the thinning network is the channel concatenation result of the initial grayscale image and the deformed image, i.e., the number of channels in the input tensor is 2C (C is the number of channels in the initial image). Each convolutional operation uses a 3×3 convolutional kernel with a stride of 1, and is accompanied by padding to maintain spatial resolution. BatchNorm layers and non-linear activation functions are used alternately between convolutional layers to improve the model's convergence speed and generalization ability. The network depth is typically set to 4-6 layers to ensure sufficiently complex feature fusion capabilities while avoiding overfitting.

[0063] In some implementations, the number of output channels of the refinement network is kept consistent with the number of input image channels, facilitating subsequent image reconstruction and micro-expression recognition tasks. The network output image, after nonlinear activation, is then fused with the original image through residual connections or skip connections to further enhance detail expression. This step establishes a strong correlation between the event data output by the event camera and the image data, enabling the reconstructed image to maintain temporal consistency while possessing higher spatial fidelity and visual clarity.

[0064] In practical applications, this refined network is particularly suitable for scenarios with low light, high dynamic range, or rapid micro-expression changes. By removing artifacts such as blurring and ghosting during motion compensation, it significantly improves the visual quality of image sequences, providing higher-quality input data for subsequent micro-expression recognition based on spiking neural networks (SNNs), thereby effectively improving recognition accuracy and robustness.

[0065] S42, the number of input channels of the image thinning network is the sum of the number of channels of the initial image and the number of channels of the deformed image, and the number of output channels is the same as the number of channels of the initial image.

[0066] Specifically, in the design of the image thinning network, the number of input channels is set to the sum of the number of channels in the initial image and the number of channels in the deformed image, while the number of output channels remains consistent with the number of channels in the initial image. The core of this design lies in improving the quality and detail fidelity of image reconstruction through multi-channel information fusion.

[0067] The input to the image thinning network consists of two parts: an initial grayscale image from the event camera, typically a single-channel image, and a deformed image obtained through motion field estimation and deformation sampling operations, with the same number of channels as the initial image. In some implementations, if the initial image is in RGB format (3 channels), the deformed image should also be 3 channels to ensure consistency in spatial structure. Therefore, the number of input channels is 1+1=2 or 3+3=6, depending on the choice of image modality. The shape of the input tensor is , where is the batch size, is the time step, is the image height, and is the image width. The thinning network typically consists of multiple convolutional layers (Conv2D), normalization layers (BatchNorm2D), and nonlinear activation functions (such as ReLU or SELU) to achieve joint optimization of static structure and dynamic deformation information.

[0068] In this invention, the initial image is typically an 8-bit grayscale image or a 24-bit RGB image, with pixel values ​​ranging from [0, 255]. Before being input into the network, it needs to be normalized to the range of [0, 1] or [-1, 1]. The number of channels in the convolutional layers of the refinement network is configured as a multiple of the number of input channels. For example, when the input has 6 channels, the first convolutional layer can be set to 64 channels to enhance feature representation. The number of output channels is strictly consistent with the number of channels in the initial image, ensuring that the final reconstructed image remains compatible with the original image in terms of channel dimension, facilitating subsequent processing by the micro-expression recognition model.

[0069] This step is widely used in micro-expression recognition systems that fuse event cameras and image frames. In low-power edge devices (such as embedded neuromorphic computing modules), the robustness and accuracy of micro-expression recognition can be effectively improved through the joint optimization of event-driven dynamic visual information and static image information, especially in highly dynamic scenes or environments with drastic changes in lighting.

[0070] The image thinning network, by fusing multi-channel information from the initial and deformed images, effectively removes artifacts during image reconstruction, enhances the texture details of facial micro-expressions, and improves the spatial resolution and temporal consistency of the image. This step, a crucial component of the image reconstruction process, provides high-quality continuous image sequences as input for subsequent micro-expression recognition based on spiking neural networks, thereby significantly improving the overall system's recognition performance and adaptability.

[0071] The image reconstruction method based on multi-time-step event data in this invention can effectively utilize the high temporal resolution of event cameras to reconstruct high-quality continuous image sequences through multi-time-step event data, thereby improving the temporal modeling accuracy and robustness of micro-expression recognition, while reducing system power consumption and data redundancy.

[0072] Furthermore, it also includes: S5 performs multimodal fusion of the reconstructed image sequence and the event frame sequence, and inputs it into a spiking neural network for temporal feature extraction and emotion classification.

[0073] Specifically, this step involves multimodal fusion of the reconstructed image sequence and the event frame sequence, and then inputting them into a spiking neural network (SNN) for temporal feature extraction and emotion classification. The core of this step is to fuse two heterogeneous visual modalities to enhance the model's ability to perceive the dynamic processes of micro-expressions.

[0074] The image sequence is generated by an image reconstruction algorithm based on multi-timestep event data, exhibiting high temporal resolution and spatial fidelity. The event frame sequence consists of ON / OFF events accumulated by the event camera within a fixed time step (T=24), reflecting the dynamic changes of local pixels. Multimodal fusion employs a channel stitching method, fusing the reconstructed image sequence (shape [B×T×H×W]) and the event frame sequence (shape [B×T×6×H×W]) along the channel dimension to form a joint input tensor of [B×T×(3+6)×H×W]. This tensor serves as the input to the SNN, extracting temporal features of the image modality and the event modality through a dual-branch structure.

[0075] Both the image encoder and the event encoder employ convolutional layers (Conv2D), batch normalization (BatchNorm), and LIF spiking neurons, with an output feature dimension of [B×T×C×H×W], where C is the number of feature channels. The membrane time constant of the LIF neurons is set to 20ms, and the threshold voltage is set to 1.0V. Bilinear interpolation and temporal consistency constraints are used to ensure the temporal stability of the spiking process. The classifier consists of two layers of fully connected spiking neurons, working in conjunction with the LIF activation function to ultimately output the sentiment category via average pooling over the time dimension.

[0076] This approach is applicable to facial micro-expression recognition systems in low-power edge devices, such as smart security, human-computer interaction, and clinical psychological assessment. By fusing the high dynamic response of event cameras with the structural information of image sequences, the model maintains high recognition robustness under complex lighting, rapid motion, and occlusion conditions.

[0077] By employing a multimodal fusion mechanism, the temporal sensitivity of event streams and the structural integrity of images are effectively integrated, providing high-quality input representations for SNNs and thus improving the accuracy and real-time performance of micro-expression recognition. Simultaneously, this method significantly reduces the bandwidth and power consumption requirements of traditional frame-driven systems, aligning with the development trends of neuromorphic computing and edge intelligence.

[0078] S6 uses the LIF neuron model for its spiking neural network and trains it using BPTT combined with the surrogate gradient method.

[0079] Specifically, in this invention, the Spiking Neural Network (SNN) employs the Leaky Integrate-and-Fire (LIF) neuron model and is trained using Back Propagation Through Time (BPTT) combined with a surrogate gradient method. This is the core technology for achieving efficient and low-power operation of the micro-expression recognition model. The LIF neuron model, by simulating the dynamic changes in membrane potential of biological neurons, possesses time-sensing capabilities and can effectively model the instantaneous dynamic features of micro-expressions.

[0080] Since the spiking process involves a non-differentiable step function, traditional gradient descent methods cannot be directly applied. Therefore, this invention employs a surrogate gradient method, using a differentiable approximation function (such as a sigmoid, Gaussian, or piecewise linear function) to replace the derivative of the step function during backpropagation, thereby achieving end-to-end training of spiking neurons. Specifically, in BPTT training, the model receives joint input of time-series event data and image data, transforming the dynamic process of the SNN into a differentiable temporal computation graph through time unfolding, allowing gradients to propagate back along the time dimension and optimize network parameters.

[0081] This step plays a crucial role in the entire technical solution. By combining the high temporal resolution of the event camera with the temporal encoding capability of LIF neurons, the model can accurately capture the instantaneous changes in micro-expressions. At the same time, it achieves low-power deployment under the heterogeneous architecture of FPGA+ARM+TPU, providing a solution for micro-expression recognition systems that combines real-time performance and energy efficiency.

[0082] Example 2 To implement the brain-like vision-based micro-expression recognition method proposed in this invention, high-precision facial and facial expression dynamic data must first be acquired using an event-like camera. The event camera can accurately capture microsecond-level visual changes and simultaneously output traditional image frames as well as synchronized ON and OFF event data. For the output event data, we perform temporal matching with the image frames and divide it into multi-time-step event frames at a fixed time step (T=24). To establish a dataset covering various facial and micro-expression states, the acquisition scheme includes the following steps: 1) Subject recruitment: Collect facial samples from volunteers of various ages, genders, and skin colors; 2) Facial Expression Induction: Using video clips, voice prompts, or mirror guidance, subjects are briefly induced to produce the following micro-expressions: happy, surprised, angry, fearful, sad, disgusted, and neutral.

[0083] 3) Acquisition process: Performed under standard ambient lighting; image frames, orientation and orientation event data are recorded synchronously, and the event data is accumulated into event frames of multiple time steps; each expression is collected 3-5 times to increase sample diversity; at the same time, information such as subject identity, expression type, and acquisition time can be recorded.

[0084] Furthermore, this section proposes an event-data-driven image sequence reconstruction method, aiming to utilize the sparse visual information captured by event cameras at high temporal resolution to reconstruct image frames within a target time period frame by frame. The overall process includes six modules: event feature extraction, motion field estimation and deformation sampling network, image deformation and fusion, image thinning and unsupervised learning, as shown in the overall framework. Figure 2 As shown.

[0085] The proposed method utilizes the high temporal precision event stream output by an event camera and combines it with the initial grayscale frame for refinement and optimization, balancing spatial structural clarity and temporal consistency to generate a high-fidelity image sequence. This provides more subtle and instantaneous facial dynamic information for subsequent emotion recognition, effectively improving the accuracy of micro-expression recognition. The specific steps are as follows: Event feature extraction. In an event vision system, each event is characterized by its spatial location. timestamp Event polarity Composition. To effectively model dynamic scene information within a continuous time period, the event stream is segmented by time, constructing a fixed number of event frame sequences:

[0086] in Indicates the first The event images at each time step. In addition, a reference grayscale image is provided to constrain the event-driven image reconstruction process. As initial background information, it is used to provide static textures and prior knowledge.

[0087] To fully extract spatiotemporal motion information from event images, we designed a set of event encoders with shared parameters. Each frame of the event image Mapped to a high-dimensional feature space representation:

[0088] in This indicates the number of channels output by the encoder. The first The event frame extracts spatial edge information, local motion cues, and event intensity distribution. To ensure temporal consistency, all event frames are processed using an encoder with the same structure. The event encoder consists of multiple convolutional layers, batch normalization, and nonlinear activation functions, possessing strong feature representation capabilities and enabling joint training with downstream modules in an end-to-end manner.

[0089] Sports field estimation. This involves obtaining the event feature sequence. Then, through a motion estimation network To generate pixel-level motion vector fields for the current time point frame by frame. Specifically, for the first... Frame feature map The corresponding two-dimensional motion field is obtained through the following mapping:

[0090] in Indicates from the initial image To the The motion vectors of the pixels in the frame image. This motion field encodes the offset of each pixel in the time dimension and can be used for subsequent image deformation and sampling operations. To ensure stability and trainability, the motion field output by the network uses... Activation function normalized to The interval facilitates addition and superposition with the standard sampling grid, and adapts to the bilinear interpolation mechanism required by the image deformation module.

[0091] Deformable sampling mesh generation. To achieve event-guided image reconstruction, we use a motion field to deformably sample the initial grayscale image, thereby generating the image frame at the target time step. This process relies on two-dimensional pixel-level motion vectors estimated from features extracted from the event encoder, which are then used to construct the deformable sampling mesh. The overall steps are as follows: 1): Standard sampling grid construction, , 2): Sports field estimation and superposition. , 3): Coordinate normalization, , 4): Dimensional rearrangement .

[0092] In one embodiment of the present invention, a standard sampling grid is constructed: Let the image space size be First, we construct a standard two-dimensional raster coordinate system and define the original coordinate position of each pixel. We then use tensor operations to create a static sampling mesh:

[0093]

[0094]

[0095] in, This represents a grid of spatial coordinates for each pixel. This grid is then expanded to batch dimensions to accommodate batch data input.

[0096] In one embodiment of the present invention, the motion field is estimated and superimposed: Based on the feature map output by the event frame encoder, a two-dimensional motion field tensor is obtained through multiple convolutional modules. The channel dimension is the pixel displacement in both directions. We overlay the sports field onto the standard grid to obtain the updated sampled grid:

[0097] in, This indicates the pixel sampling position after being affected by the event-guided displacement.

[0098] In one embodiment of the present invention, coordinate normalization is performed: To accommodate bilinear sampling operations in deep learning frameworks, the sampling grid needs to be normalized to... scope:

[0099]

[0100] Normalized sampling grid It can be directly input into the grid_sample function to achieve spatial transformation of the original image.

[0101] In one embodiment of the present invention, dimensional rearrangement is performed: Finally, to meet the input dimension requirements of grid_sample, the sampling grid was changed from... Convert to .

[0102] Image deformation and fusion. Using a normalized sampling grid, we can sample the initial image using bilinear interpolation:

[0103] in Representing the reconstructed image of the moving motion field, grid_sample automatically completes the interpolation process, thus obtaining the continuous spatial transformation of the image.

[0104] Image refinement. This involves obtaining the pixel-level motion field at each time step. Then, this section utilizes a deformable sampling mechanism to process the initial grayscale image. Perform spatial transformation to generate the current time point. Preliminary reconstructed image This process employs bilinear interpolation sampling to ensure the continuity and differentiability of the deformation. The specific deformation operation is defined as follows:

[0105] in Indicates based on sports field The constructed normalized sampling grid is compatible with the bilinear sampling function grid_sample of deep learning frameworks such as PyTorch.

[0106] To further improve image quality, this invention introduces a thinning network to refine the initial image. With deformed images The images are stitched together along the channel dimension and then fed into the network to generate the final optimized reconstructed image. :

[0107] in This represents a network for refining grayscale images, consisting of multiple layers of convolutions, normalization, and activation functions. Its purpose is to fuse static structural and dynamic motion information, removing reconstruction artifacts and enhancing texture details. Finally, repeating this process gradually generates a complete sequence of consecutive grayscale images.

[0108] Loss function design. To effectively constrain the temporal image reconstruction process, this invention designs a multi-component unsupervised loss function system, mainly including the following four components: 1) Time consistency loss: To constrain the brightness continuity between adjacent frames, a L2 norm is introduced to measure the difference between adjacent frames:

[0109] This encourages the network to generate reconstructed images that are smooth over time.

[0110] 2) Event consistency loss: To enhance the consistency between the reconstructed image and the event trigger, the mean square error between the reconstructed image and the average brightness map of the event frame is calculated:

[0111] in For the first Average channel grayscale of the frame event image.

[0112] 3) Structural consistency loss: The structural similarity index is used to measure the structural fidelity between the reconstructed image and the original image:

[0113] This approach emphasizes the similarity of local brightness, contrast, and structure, which helps preserve the texture information of the reconstructed image.

[0114] 4) Gradient consistency loss: To enhance image edge details and restore object contours, a gradient matching term is introduced:

[0115] in , These represent the first-order gradient operators in the horizontal and vertical directions, respectively.

[0116] Finally, the total loss function is expressed as a weighted summation:

[0117] The weights for each item are set as follows: =0.1, =0.3, =0.3, =0.3.

[0118] Network Training and Optimization Strategies. To train the entire image reconstruction model, this section uses the Adam optimizer for end-to-end training, setting the following parameters: Initial learning rate: 0.001; Weight decay: 0.0001; Momentum parameter: .

[0119] During training, the following strategies were used to improve model performance: the learning rate was reduced to 0.1 times its original value every 20 epochs; training was stopped early if the validation set loss did not decrease within 10 consecutive epochs; all input images and event frames were uniformly normalized to the range of [0,1] or [-1,1]; peak signal-to-noise ratio and structural similarity were used as validation metrics to select the best model for testing.

[0120] Furthermore, this section proposes a multimodal micro-expression recognition algorithm based on a Spiking Neural Network (SNN). This model integrates image data and event-driven visual data, utilizing event motion data and spatial structure information of the image to improve the model's adaptability and robustness to complex environments such as occlusion and lighting changes. While preserving temporal dynamic information, it significantly reduces computational overhead, making it suitable for low-power edge devices. Structurally, this method employs a dual-branch temporal coding approach, multi-stage fusion extraction, and a temporal pulse classifier to achieve multimodal micro-expression recognition.

[0121] The overall framework of the model is as follows Figure 3 As shown, the input includes two modalities: image sequences and event data streams. After feature extraction by two parallel encoders, the input is fused in the temporal dimension, followed by multi-layer feature extraction to further extract high-level spatiotemporal information, and finally outputs the emotion category through a pulse fully connected classifier. Feature extraction. The image data is a reconstructed image sequence with a shape of... ,in For batch size, Indicates the time step. These represent the height and width of the image, respectively. This modality is used to provide static facial or scene emotional cues. The event data is a dual-view event stream, with input dimensions of [missing dimension]. , where 2 represents direction and The event data dimension in the direction. Each frame is processed by channel merging to obtain a 6-channel event image, which is used to capture high-frequency micro-expression or movement changes.

[0122] The model employs a dual-branch structure to temporally encode the image modality and the event modality separately. The image encoder structure includes convolutional layers, BatchNorm layers, spiking neurons, and average pooling layers, providing time-aware capabilities. The output shape is... The event encoder has the same structure as the image encoder, but its input is 6 channels, capturing temporal event features. The output shape is... .

[0123] The spiking neurons in the network adopt the classic LIF neuron model. The dynamic model of LIF can be described by three equations: charging, discharging, and resetting. The charging equation is as follows:

[0124] in The membrane time is a constant. This represents the neuron membrane voltage at the previous moment. It is the external input at the current moment.

[0125] When the instantaneous membrane potential exceeds the threshold If yes, a pulse is fired; otherwise, no pulse is fired. The neuron firing process is as follows:

[0126] in For step functions:

[0127] When the spiking neuron fires a pulse, the voltage resets to... When the spiking neuron does not fire, the voltage remains unchanged. The reset equation for the spiking neuron is as follows:

[0128] in This represents the membrane potential at the current moment.

[0129] Feature fusion and multi-stage extraction. After image and event modality encoding, they are concatenated along the channel dimension to obtain... The model then uses a three-stage temporal convolutional block for high-level semantic extraction. This multi-stage extraction process refines information layer by layer while maintaining a time-aware mechanism, enabling the model to model spatiotemporal variations in emotional expression.

[0130] Global temporal feature extraction and classifier design. After three-stage convolution, the output features are: To reduce the number of parameters and memory requirements, the model employs global average pooling, compressing the spatial features at each time step into a single 32-dimensional vector:

[0131] The final shape is The model extracts temporal global features. Finally, a spiking classifier consisting of two fully connected neurons is used in conjunction with LIF activation to achieve dynamic discrimination. The model ultimately outputs an average pooled value over the time dimension.

[0132] Loss function and training strategy. This model is trained using cross-entropy loss and is suitable for various sentiment recognition tasks.

[0133] in For the number of emotion categories, To predict probabilities for the model, One-hot encoding of the true labels is used. The training process employs the Adam optimizer with an initial learning rate of 1e-3, followed by StepLR to decrease the learning rate. The loss function and accuracy are used to evaluate model performance, supporting metrics such as confusion matrix and F1-score.

[0134] The hardware implementation of this invention is as follows: Neuromorphic heterogeneous computing micromodules are an advanced computing architecture that employs a unique heterogeneous hardware architecture combining FPGA (Field-Programmable Gate Array), ARM (Advanced Relational Architecture), and TPU (Tensor Processing Unit). This architecture cleverly integrates the performance advantages of each chip, enabling it to achieve high frame rates, high dynamic range, and low latency response capabilities, efficiently processing various complex data and tasks. Figure 4 The module functional block diagram shown is shown.

[0135] In this module, the KA200 chip serves as the main processor, undertaking core computing and control tasks. It boasts powerful processing capabilities and efficient energy management, enabling it to rapidly process large amounts of data and allowing for flexible configuration and optimization based on diverse task requirements. The KA200's high performance provides a solid foundation for the stable operation of the entire module.

[0136] The FPGA plays a crucial role in this module, serving as an extension for the video and control interfaces. Through the FPGA, video image acquisition and peripheral device control can be achieved. The programmability of the FPGA allows for customized configurations to meet various complex interface requirements based on different application scenarios. In video image acquisition, the FPGA enables high-precision, high-speed data acquisition, providing a high-quality data source for subsequent image processing and analysis. Regarding the control interface, the FPGA can flexibly communicate and control various peripheral devices, achieving precise control of the entire system.

[0137] This brain-inspired heterogeneous computing micro-module design fully draws on the neural network structure and information processing mechanisms of the human brain. Through the collaborative work of different chips, it achieves highly efficient information processing capabilities similar to the human brain. This module has broad application prospects in fields such as artificial intelligence, machine vision, and intelligent control. For example, in intelligent monitoring systems, it can quickly process video image data to achieve functions such as target recognition and behavior analysis; in intelligent robots, it can achieve precise control of robot movement and environmental perception; in intelligent transportation systems, it can monitor and analyze traffic flow in real time, providing strong support for traffic management.

[0138] In summary, neuromorphic heterogeneous computing micromodules, with their unique architecture and high performance, offer a new approach and solution for the development of modern intelligent computing. They organically integrate the advantages of different chips, achieving high frame rates, high dynamic range, and low latency response capabilities, thus meeting the needs of various complex applications. With continuous technological development and innovation, these modules will undoubtedly be widely used in more fields, bringing greater convenience and intelligent experiences to people's lives and work.

[0139] The specifications of the heterogeneous hardware system are shown in Table 1: Table 1

[0140] The dimensions of the neuromorphic heterogeneous computing micro-module structure are no greater than 40mm (length) × 40mm (height) × 40mm (thickness). The module structure's shape is as follows: Figure 5 As shown.

[0141] The micro-expression recognition method based on brain-like vision of the present invention can effectively utilize the high temporal resolution characteristics of event cameras to reconstruct high-quality continuous image sequences through multi-time step event data, thereby improving the temporal modeling accuracy and robustness of micro-expression recognition, while reducing system power consumption and data redundancy.

[0142] Example 3 To achieve the above embodiments, such as Figure 6 As shown, this embodiment also provides a micro-expression recognition device 10 based on brain-like vision, including: The event frame construction module 100 is used to divide the event stream output by the event camera into fixed time steps, construct a multi-time step event frame sequence, and synchronously acquire the initial grayscale image as a static texture prior. The event feature extraction module 200 is used to extract high-dimensional features of event frames at each time step through an event encoder with shared parameters, and generate corresponding pixel-level motion vector fields. The image deformation sampling module 300 is used to perform deformation sampling on the initial grayscale image using the motion vector field to generate a reconstructed image at the target time step; The image fusion optimization module 400 is used to stitch the initial grayscale image and the deformed image in the channel dimension, input the image thinning network for fusion optimization, and output a continuous grayscale image sequence.

[0143] Furthermore, the event frame building module is also used for: The event stream is segmented according to a preset time step to generate event frames with multiple time steps; Align the event frame with the synchronously acquired initial grayscale image in the time dimension to ensure that the timestamp error between the event frame and the image frame is less than a preset time threshold.

[0144] Furthermore, the event feature extraction module is also used for: The event encoder consists of multiple convolutional layers, batch normalization layers, and nonlinear activation functions, used to extract spatial edge information, local motion cues, and event intensity distribution of event frames. The motion vector field output by the motion field estimation network is normalized to the interval [-1, 1] using the tanh activation function to accommodate subsequent bilinear interpolation operations.

[0145] Furthermore, the image fusion optimization module is also used for: The image thinning network consists of multiple convolutional layers, normalization layers, and nonlinear activation functions, and is used to remove artifacts in the reconstructed image and enhance texture details; The number of input channels for the image thinning network is the sum of the number of channels in the initial image and the number of channels in the deformed image, and the number of output channels is the same as the number of channels in the initial image.

[0146] Furthermore, it also includes: The multimodal fusion module is used to perform multimodal fusion of reconstructed image sequences and event frame sequences, and inputs them into a spiking neural network for temporal feature extraction and emotion classification. The emotion classification module uses a LIF neuron model and is trained using BPTT combined with a surrogate gradient method.

[0147] The micro-expression recognition device based on brain-like vision in this invention can effectively utilize the high temporal resolution of event cameras to reconstruct high-quality continuous image sequences through multi-time-step event data, thereby improving the temporal modeling accuracy and robustness of micro-expression recognition, while reducing system power consumption and data redundancy.

[0148] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples" refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0149] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two or three, unless otherwise explicitly specified.

Claims

1. A micro-expression recognition method based on brain-like vision, characterized in that, include: S1, divide the event stream output by the event camera into fixed time steps, construct a multi-time step event frame sequence, and synchronously acquire the initial grayscale image as a static texture prior; S2, high-dimensional features of event frames at each time step are extracted by an event encoder with shared parameters to generate corresponding pixel-level motion vector fields; S3, use the motion vector field to deform and sample the initial grayscale image to generate the reconstructed image of the target time step; S4 concatenates the initial grayscale image and the deformed image along the channel dimension, and the input image thinning network performs fusion optimization to output a continuous grayscale image sequence.

2. The method as described in claim 1, characterized in that, S1 includes: S11, the event stream is segmented according to a preset time step to generate event frames with multiple time steps; S12, align the event frame with the synchronously acquired initial grayscale image in the time dimension to ensure that the timestamp error between the event frame and the image frame is less than a preset time threshold.

3. The method as described in claim 1, characterized in that, The S2 includes: S21, the event encoder consists of multiple convolutional layers, batch normalization layers and nonlinear activation functions, used to extract spatial edge information, local motion cues and event intensity distribution of event frames; S22, the motion vector field output by the motion field estimation network is normalized to the interval [-1, 1] using the tanh activation function to adapt to the subsequent bilinear interpolation operation.

4. The method as described in claim 1, characterized in that, The S4 includes: S41, the image thinning network consists of multiple convolutional layers, normalization layers and nonlinear activation functions, used to remove artifacts in the reconstructed image and enhance texture details; S42, the number of input channels of the image thinning network is the sum of the number of channels of the initial image and the number of channels of the deformed image, and the number of output channels is the same as the number of channels of the initial image.

5. The method as described in claim 1, characterized in that, Also includes: S5 performs multimodal fusion of the reconstructed image sequence and event frame sequence, and inputs it into a spiking neural network for temporal feature extraction and emotion classification; S6 uses the LIF neuron model for the spiking neural network and trains it using BPTT combined with the surrogate gradient method.

6. A micro-expression recognition device based on brain-like vision, characterized in that, include: The event frame construction module is used to divide the event stream output by the event camera into fixed time steps, construct a multi-time step event frame sequence, and synchronously acquire the initial grayscale image as a static texture prior. The event feature extraction module is used to extract high-dimensional features of event frames at each time step through an event encoder with shared parameters, and generate corresponding pixel-level motion vector fields. The image deformation sampling module is used to perform deformation sampling on the initial grayscale image using the motion vector field to generate a reconstructed image at the target time step; The image fusion optimization module is used to stitch the initial grayscale image and the deformed image together in the channel dimension. The input image thinning network performs fusion optimization and outputs a continuous grayscale image sequence.

7. The apparatus as claimed in claim 6, characterized in that, The event frame construction module is also used for: The event stream is segmented according to a preset time step to generate event frames with multiple time steps; Align the event frame with the synchronously acquired initial grayscale image in the time dimension to ensure that the timestamp error between the event frame and the image frame is less than a preset time threshold.

8. The apparatus as claimed in claim 6, characterized in that, The event feature extraction module is also used for: The event encoder consists of multiple convolutional layers, batch normalization layers, and nonlinear activation functions, used to extract spatial edge information, local motion cues, and event intensity distribution of event frames. The motion vector field output by the motion field estimation network is normalized to the interval [-1, 1] using the tanh activation function to accommodate subsequent bilinear interpolation operations.

9. The apparatus as claimed in claim 6, characterized in that, The image fusion optimization module is also used for: The image thinning network consists of multiple convolutional layers, normalization layers, and nonlinear activation functions, and is used to remove artifacts in the reconstructed image and enhance texture details; The number of input channels for the image thinning network is the sum of the number of channels in the initial image and the number of channels in the deformed image, and the number of output channels is the same as the number of channels in the initial image.

10. The apparatus as claimed in claim 6, characterized in that, Also includes: The multimodal fusion module is used to perform multimodal fusion of reconstructed image sequences and event frame sequences, and inputs them into a spiking neural network for temporal feature extraction and emotion classification. The emotion classification module uses a LIF neuron model and is trained using BPTT combined with a surrogate gradient method.

Images