Human posture recognition method based on multimodal neural network framework

Abstract

The present invention relates to the fields of computer vision and neural network processing technology, and in particular to a method for human posture recognition based on a multimodal neural network framework. The method comprises: collecting event data and representing the event data using a five-channel event representation method; dividing the preprocessed five-channel event data into three channels: spatial, event feature, and point cloud channels; performing feature extraction after channel splitting, and optimizing and integrating features from different modalities through dynamic feature enhancement and cross-modal fusion; employing a skeleton constraint module and a spatiotemporal joint processing module to enhance the network's understanding of human structure and model spatiotemporal dependencies; employing a prediction head module to output the final coordinates of human key points; and employing a multi-task learning framework for training and optimization to obtain posture recognition results. The method's advantages include effectively processing the sparse and asynchronous nature of event data, resolving the "blind spot problem" of static limbs, maintaining low power consumption, and achieving high-precision human posture estimation.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and neural network processing, and in particular to a human posture recognition method based on a multimodal neural network framework. Background Art

[0002] An event camera is a biomimetic vision sensor. Unlike traditional cameras that capture full-frame images at a fixed frequency, an event camera asynchronously generates events only when pixel brightness changes exceed a preset threshold. Each event is represented as a four-tuple e = (x, y, t, p), where (x, y) is the pixel location, t is a microsecond timestamp, and p is the polarity (indicating an increase or decrease in brightness). This operating mode gives event cameras advantages such as microsecond temporal resolution, high dynamic range (120dB vs. 60dB), and low power consumption, making them particularly suitable for processing high-speed motion and complex lighting scenes.

[0003] However, the sparse and asynchronous nature of event data also presents processing challenges. Each individual event carries very little contextual information and must be aggregated before it can be used in downstream tasks. Current event data processing methods fall into two main categories: sparse representation and dense representation. Sparse representation preserves the original characteristics of event data but is limited by the lack of specialized hardware and efficient algorithms. Dense representation, on the other hand, converts events into a format compatible with traditional computer vision, facilitating the application of mature deep learning techniques.

[0004] In the field of human pose estimation, event cameras are attracting increasing attention due to their advantages. Researchers have proposed the DHP19 dataset, which uses multiple synchronized event cameras to record human motion. They use a CNN to estimate 2D keypoints from event frames, and then triangulate the 3D pose. Others have proposed the Temporally Ordered Volume of Recent Events (TORE) representation, which improves pose estimation accuracy by storing the raw event timing information. Others have proposed LiftMonoHPE, which uses a monocular event camera to estimate 3D pose, first predicting 2D keypoints and then recovering the 3D information using a deep learning model. Others have attempted to process events as point cloud data, proposing the EventPointPose model, but this model fails to fully exploit the temporal nature of events.

[0005] In terms of integrating event data with neural networks, some researchers have proposed a hybrid ANN-SNN architecture. This architecture uses a low-frequency ANN to initialize the state of a high-frequency SNN, addressing the slow convergence and state decay issues of SNNs. Others have designed a hybrid network combining SNNs and ANNs for event-driven optical flow estimation, where the SNN handles event encoding and the ANN is responsible for feature extraction and prediction.

[0006] However, existing methods for human pose estimation based on event cameras face multiple technical challenges, severely limiting their effectiveness in practical applications. Mainstream methods accumulate events into two-dimensional representations, such as event frames, time surfaces, or voxel grids. This inevitably loses the precise temporal information inherent in the event data, a significant problem when processing fast, complex movements. Furthermore, event cameras only generate events when brightness changes, leading to a "blind spot problem" in static regions: when a part of the human body remains stationary, no events are generated in that region and the representation disappears, resulting in incomplete pose estimation. Furthermore, existing methods often use fixed time windows for event stacking, which can lead to information redundancy or loss when the event rate is too high or too low. While methods that directly process event streams, such as spiking neural networks, can preserve temporal characteristics, they suffer from slow convergence, state decay, and low computational efficiency. Methods that convert events into dense representations, while computationally efficient, are limited in accuracy and fail to fully exploit the advantageous properties of event cameras. More importantly, most existing methods mix different event characteristics (such as spatial location, timestamp, and polarity), failing to effectively distinguish and utilize this heterogeneous information, resulting in inefficient feature extraction. Furthermore, most methods fail to effectively utilize prior knowledge of the human skeleton and lack the necessary skeletal structural constraints, resulting in pose prediction in complex scenes often not conforming to human anatomy. In summary, the main limitations of existing methods are as follows: First, methods that accumulate events into two-dimensional frames inevitably lose precise temporal information. Second, while methods that directly process event streams, such as spiking neural networks (SNNs), preserve temporal characteristics, they are difficult to train, have limited accuracy, and suffer from slow convergence and state decay. Third, static limb scenes suffer from "blind spots," where static regions do not generate events and therefore disappear from the representation.

[0007] Currently, there are several related solutions for event-based data stream representation methods both domestically and internationally. In the 2021 paper "Exploring Event-based Human Pose Estimation with 3D Event Representations," author Jing Zhao published in the Conference on Computer Vision and Pattern Recognition (CVPR) journal. The paper proposed a voxel-based multi-scale Transformer network (VMST-Net), which improves information extraction efficiency by constructing event voxels in 3D space. This method attempts to capture both spatial and temporal information about events, but it is computationally complex and still does not address the issue of missing information in static regions. Furthermore, it still faces stability issues in rapidly changing motions and complex environments.

[0008] In summary, although existing methods have made progress in the field of event-driven human pose estimation, there is still a lack of a comprehensive solution that can effectively handle the spatiotemporal characteristics of events, solve the problem of missing information in static areas, and maintain low computational complexity. Summary of the Invention

[0009] In order to solve the above problems, the present invention provides a human posture recognition method based on a multimodal neural network framework.

[0010] The present invention aims to provide a method for human posture recognition based on a multimodal neural network framework, which specifically comprises the following steps:

[0011] S1. Data preprocessing: Event data is collected and represented using a five-channel event representation method. The five-channel event representation method is as follows:

[0012] E'=(x,y,T,P,E);

[0013] Where x and y are pixel positions, T is the average timestamp, P is the cumulative positive, and E is the event count;

[0014] S2. Channel Splitting and Trimodal Separation: The five-channel event data preprocessed in step S1 is divided into three channels: a spatial channel containing normalized spatial coordinates (x, y); an event feature channel containing temporal and polarity features (T, P, E); and a point cloud channel: the complete five-channel data is treated as point cloud data, preserving the original three-dimensional characteristics of the event.

[0015] S3. Multimodal feature extraction: After completing channel splitting, feature extraction is performed on the data of the spatial channel, event characteristic channel, and point cloud channel respectively;

[0016] S4. Dynamic Feature Enhancement and Cross-modal Fusion: After feature extraction, dynamic feature enhancement and cross-modal fusion are used to optimize and integrate features from different modalities.

[0017] S5. Skeleton Constraint and Spatiotemporal Processing: After feature fusion, the skeleton constraint module and spatiotemporal joint processing module are used to enhance the network's understanding of human structure and spatiotemporal dependencies.

[0018] S6. Keypoint prediction and training: Use the prediction head module to output the final coordinates of the human body keypoints;

[0019] S7. Using the preprocessed data from step S1, repeat steps S2 to S6 for training. The training method includes a three-level loss function and uses a substitute gradient method for back propagation to achieve end-to-end training to obtain posture recognition results.

[0020] Preferably, step S3 specifically includes:

[0021] S301. Processing spatial channel data using a multi-gradient feature extraction module;

[0022] S302. Processing event characteristic channel data using a pulse feature processor;

[0023] S303. Use a hierarchical point cloud encoding module to perform point-level feature extraction, local area feature extraction, and global feature extraction on the point cloud data.

[0024] Preferably, the multi-gradient feature extraction module in step S301 includes four parallel branches, namely, a small-scale feature extraction convolution kernel, a medium-scale feature extraction convolution kernel, a large-scale feature extraction convolution kernel and an ultra-large-scale feature extraction convolution kernel, and the convolution kernel sizes are 1×1, 3×3, 5×5 and 7×7 respectively; the output of each branch is spliced and fused after batch normalization and activation function processing.

[0025] Preferably, the point-level feature extraction in step S303 processes each point independently through a multi-layer perceptron; the local area feature extraction captures the relationship between the point and its k nearest neighbors; and the global feature extraction obtains the overall structural information through a maximum pooling operation.

[0026] Preferably, the pulse feature processor in step S302 includes an adaptive LIF neuron or an Izhikevich neuron; the dynamic equation of the adaptive LIF neuron is as follows:

[0027]

[0028] Membrane potential update (discretization):

[0029]

[0030] Threshold Update:

[0031]

[0032] Pulse Generation:

[0033]

[0034] Where spike is the action potential of the neuron, v is the membrane potential, θ is the adaptive threshold, and τ mem is the membrane potential time constant, τ thresh is the threshold time constant, θ rest is the resting threshold, I is the input current, and v t represents the membrane potential at time t, θ t represents the adaptive threshold at time t, I t represents the input current at time t, and Heaviside is a step function;

[0035] The dynamic equation of the Izhikevich neuron is:

[0036]

[0037] Pulse trigger condition: if v ≥ threshold, then spike = 1, otherwise spike = 0; reset after pulse: if spike = 1, then v = c, u = u + d;

[0038] Where v is the membrane potential, u is the membrane potential recovery variable, I is the input current, and a, b, c, and d are parameters that control neuronal behavior.

[0039] Preferably, the pulse feature processor in step S302 also includes a time gating unit and a bidirectional LSTM layer to further extract timing features and long-term dependencies; the specific operations are as follows: the event feature channel (T, P, E) data is convolutionally preprocessed through a convolution pulse encoder; the convolved features are processed using an adaptive LIF neuron model to enhance the processing capability of timing information; the adaptively processed features are time-series modeled through a bidirectional LSTM layer to capture the forward and backward dependencies of the time series; the processing capability of time information is further enhanced through the time gating unit to control the flow of information.

[0040] Preferably, step S4 specifically includes:

[0041] S401. Dynamic Feature Enhancement: The dynamic feature enhancement module uses channel attention and spatial attention mechanisms to adaptively enhance event features and spatial features based on the statistical information of input features to enhance key features;

[0042] S402. Cross-modal fusion: Use a cross-modal adaptive fusion module to perform cross-modal fusion of features.

[0043] Preferably, the mathematical representation of the channel attention mechanism in step S401 is as follows:

[0044] ;

[0045] The mathematical representation of the spatial attention mechanism is as follows:

[0046]

[0047] Where F is the input feature map, AvgPool is the global average pooling, MaxPool is the global maximum pooling, W1 and W2 are weight matrices, δ is the ReLU activation function, σ is the Sigmoid activation function, and [;] represents splicing in the channel dimension;

[0048] In step S402, the cross-modal adaptive fusion module implements adaptive fusion through the following steps:

[0049] S4021. Modality-Specific Encoding: Spatial and event features are processed separately through dedicated encoders to enhance their respective representation capabilities.

[0050] S4022. Adaptive Weight Generation: Generate dynamic weights through global pooling and multi-layer perceptron to achieve dynamic weighting between modalities. The weight calculation formula is as follows:

[0051] ;

[0052] S4023. Weighted Fusion: Weight the features of different modalities according to the generated weights;

[0053] ;

[0054] S4024. Cross-modal Attention Interaction: Deep Interaction between Modalities via Multi-Head Self-Attention Mechanism;

[0055] ;

[0056] in, F weighted Represents the feature fusion result after weighting. F s and F e are spatial features and event features respectively, w=[w1, w2] is the weight of the two modalities, Q, K, V are query, key, and value matrices, is the scaling factor.

[0057] Preferably, step S5 specifically includes the following operations:

[0058] S501. Skeleton Constraint Module: Combines spiking neural networks with graph convolution, introducing the spiking neuron mechanism into the standard graph convolution operation. The process is as follows:

[0059] First perform the standard graph convolution operation: ; Input the result Z into the pulse neuron: ;

[0060] Where X is the input feature, W is the weight matrix, b is the bias, Ã = A + I is the adjacency matrix with self-connection added, D̃ is the degree matrix of Ã, and SpikeNeuron( ) is the spike activation function;

[0061] In order to enhance the rationality of human body structure, a bone length consistency loss function is designed:

[0062] ;

[0063] Among them, bone is the line segment connecting two key point joints in the human skeleton model, bones is all the connecting line segments that constitute the complete human skeleton, length(bone) is the length of the bone, and std represents the standard deviation;

[0064] S502. Use a spatiotemporal joint processing module to capture spatiotemporal dependencies in event data; the spatiotemporal joint processing module includes multi-scale temporal coding and multi-head spatiotemporal attention;

[0065] The multi-scale temporal coding adopts a hybrid feature extraction method that combines a parallel convolution structure with frequency domain analysis. The parallel convolution structure includes convolution kernels of different sizes for processing the time dimension. The frequency domain analysis uses fast Fourier transform to extract frequency domain features. The expression is as follows:

[0066]

[0067] Among them, F is the input feature, F time is the time domain feature, F freq is the frequency domain feature, F out is the output feature, Conv k represents the convolution with kernel size k, [;] is the splicing in the channel dimension, W fusion is fused convolution, FFT is fast Fourier transform, LN is layer normalization, and g is a learnable gating parameter;

[0068] Multi-head spatiotemporal attention establishes long-distance dependencies between different spatiotemporal positions through the self-attention mechanism. The formula is as follows:

[0069]

[0070] in, F attn is the attention output, G spatial It is spatial gating, G temporal is time gating, and ⊙ is element-wise multiplication.

[0071] Preferably, the prediction formula of the prediction head module in step S6 is as follows:

[0072]

[0073] Among them, x pred 、y pred is the predicted joint point coordinate, conf is the confidence, σ is the Sigmoid activation function, F global Represents global features,Project x 、 Project y 、 Project conf are the coordinate and confidence predictors respectively, W and H are the width and height of the output feature map resolution;

[0074] The loss function is expressed as follows:

[0075] ;

[0076] in, L pose is the keypoint location loss, which uses the mean squared error to calculate the difference between the predicted keypoint and the true keypoint; λ bone represents the weight coefficient of the skeleton constraint loss, L bone is the skeleton constraint loss, which ensures that the predicted bone lengths meet physical constraints; λ temp represents the weight coefficient of temporal consistency loss, L temp is a temporal consistency loss that encourages smooth transitions between consecutive frames.

[0077] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0078] (1) Advantages of multi-dimensional information utilization: Through the channel splitting strategy and specially designed processor, the comprehensive utilization of spatial, temporal and structural information is achieved, significantly improving the accuracy of pose estimation;

[0079] (2) Static region characterization capability: The dedicated pulse feature processor of the present invention effectively solves the "blind spot problem" of traditional event processing methods in the static region by maintaining the membrane potential state of neurons;

[0080] (3) Robustness improvement: The skeleton constraint module of the present invention significantly improves the robustness of the model in partial occlusion, complex background and multi-person scenes by introducing prior knowledge of human anatomy;

[0081] (4) Dynamic adaptability: The cross-modal adaptive fusion module of the present invention can dynamically adjust the weights of different modalities based on the characteristics of the input data, allowing the network to adapt to various scenarios and motion patterns. This design performs well in handling drastically changing speed and lighting conditions and improves stability compared to fixed-weight methods.

[0082] (5) Temporal consistency: Through multi-scale temporal encoding and spatiotemporal attention mechanism, the temporal dependencies in continuous action sequences can be better captured, reducing jitter and incoherence problems.

[0083] In summary, the present invention proposes a novel hybrid pulse-point cloud neural network architecture, which separates event data into spatial channels, event feature channels, and point cloud channels through an innovative channel splitting strategy. This method combines a multi-gradient feature extraction module, a dedicated pulse feature processor, and a hierarchical point cloud encoder to effectively capture multi-scale spatial features and temporal dynamic characteristics. The system also introduces a cross-modal adaptive fusion mechanism to dynamically adjust the weights of different modalities, as well as a skeleton constraint module to constrain the prediction results using prior knowledge of human anatomy. Compared with the existing technology, the present invention can more effectively process the sparse asynchronous characteristics of event data, effectively solve the "blind spot problem" of static limbs, and achieve high-precision human posture estimation while maintaining low power consumption. It is particularly suitable for resource-constrained edge computing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a schematic diagram of an event camera.

[0085] Figure 2 This is a flowchart of a human posture recognition method based on a multimodal neural network framework provided according to an embodiment of the present invention.

[0086] Figure 3 1 is an overall network structure diagram of a multimodal neural network framework provided according to an embodiment of the present invention.

[0087] Figure 4 2 is a schematic diagram of five-channel representation and channel splitting provided according to an embodiment of the present invention.

[0088] Figure 5 2 is a schematic diagram of a multi-gradient feature extraction module, a pulse feature processor, and a hierarchical point cloud encoding module for multimodal feature extraction provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0089] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, identical modules are denoted by identical reference numerals. In the case of identical reference numerals, their names and functions are also identical. Therefore, their detailed description will not be repeated.

[0090] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0091] The novel neural network architecture proposed in this paper aims to address the above challenges through an innovative channel splitting strategy and multimodal processing method, providing a more efficient and accurate solution for event-driven human pose estimation.

[0092] The present invention provides a human posture recognition method based on a multimodal neural network framework, which specifically includes the following steps:

[0093] S1. Data preprocessing: Collect event data and represent it using the five-channel event representation method; specifically, it includes:

[0094] The raw event stream is obtained from the event camera. Each event is represented as a four-tuple: e = (x, y, t, p), where (x, y) is the pixel position, t is the microsecond timestamp, and p is the polarity value (0 or 1). The schematic diagram of the event camera is shown in Figure 1 As shown;

[0095] The event data is represented using a five-channel event representation method as follows: E'=(x,y,T,P,E);where x and y are pixel positions, T is the average timestamp, P is the cumulative positive, and E is the event count.

[0096] This step provides a unified data format for subsequent channel splitting and feature extraction. Based on the characteristics of the event camera, an innovative five-channel event representation method is designed. This representation method has the following advantages: it effectively separates spatial information and event characteristic information; it retains the key statistical characteristics of the event; it significantly reduces the amount of data and improves processing efficiency.

[0097] S2. Channel Splitting and Trimodal Separation: The five-channel event data preprocessed in step S1 is divided into three channels, as follows:

[0098] Spatial channel: contains normalized spatial coordinates (x, y), providing information about the position of the human body on the image plane;

[0099] Event feature channel: contains time and polarity features (T, P, E), providing information about the intensity and direction of the action;

[0100] Point cloud channel: Treats the complete five-channel event data as point cloud data, preserving the original three-dimensional characteristics of the event.

[0101] This step divides the data into three channels, providing clear data input for the subsequent feature extraction module, enabling the network to use specially designed processing modules for different types of information, thereby more effectively extracting and utilizing the multi-dimensional features contained in the event data.

[0102] S3. Multimodal feature extraction: After completing channel splitting, feature extraction is performed on the data of the spatial channel, event characteristic channel, and point cloud channel respectively; specifically, it includes:

[0103] S301. Processing spatial channel (x, y) data using a multi-gradient feature extraction module;

[0104] Specifically, the multi-gradient feature extraction module captures spatial features of different ranges through multi-scale parallel convolution branches. The multi-gradient feature extraction module contains four parallel branches: small-scale feature extraction convolution kernel, medium-scale feature extraction convolution kernel, large-scale feature extraction convolution kernel, and ultra-large-scale feature extraction convolution kernel. The convolution kernel sizes are 1×1, 3×3, 5×5, and 7×7, respectively. The outputs of each branch are processed by batch normalization and activation function before being spliced and fused.

[0105] This design (feature pyramid multi-scale receptive field module) enables the network to focus on both local details and global structures, and is particularly suitable for processing complex spatial relationships between human skeletal joints.

[0106] S302. Processing event characteristic channel (T, P, E) data using a pulse feature processor; the pulse feature processor includes an adaptive LIF (Leaky Integrate-and-Fire) neuron or an Izhikevich neuron; the specific operations are as follows:

[0107] Convolution preprocessing is performed on the event characteristic channel (T, P, E) data; specifically, the signal features are extracted through the convolution operation by the convolution pulse encoder;

[0108] The adaptive LIF neuron model is used to process the convolutional features and enhance the ability to process temporal information. Specifically, the adaptive LIF neuron dynamically adjusts its threshold or other parameters according to the characteristics of the input signal to better adapt to different input modes and capture the time dependency and dynamic characteristics of the signal.

[0109] The bidirectional long short-term memory (LSTM) network (Bi-LSTM) is used to perform temporal modeling on the adaptively processed features to capture the forward and backward dependencies of the time series. The time gating mechanism is used to further enhance the processing capability of temporal information, control the flow of information, and avoid problems such as gradient vanishing or exploding.

[0110] Specifically, the dynamic equation of the adaptive LIF neuron is as follows:

[0111]

[0112] Membrane potential update (discretization):

[0113]

[0114] Threshold Update:

[0115]

[0116] Pulse Generation:

[0117]

[0118] Among them, spike refers to the action potential emitted by neurons, which is the basic signal unit in spiking neural networks (SNNs); v is the membrane potential, θ is the adaptive threshold, and τ mem is the membrane potential time constant, τ thresh is the threshold time constant (determines how fast the threshold decays), θ rest is the resting threshold, I is the input current, and v t represents the membrane potential at time t, θ t represents the adaptive threshold at time t, I t Represents the input current at time t, and Heaviside is a step function.

[0119] Specifically, the dynamic equation of the Izhikevich neuron is:

[0120]

[0121] Pulse trigger condition: if v ≥ threshold, then spike = 1, otherwise spike = 0; reset after pulse: if spike = 1, then v = c, u = u + d;

[0122] Where v is the membrane potential, u is the membrane potential recovery variable, I is the input current, and a, b, c, and d are parameters that control neuronal behavior.

[0123] In addition, the pulse feature processor also includes a time gating unit and a bidirectional LSTM layer to further extract timing features and long-term dependencies; this design effectively solves the "blind spot problem" of static areas in traditional methods, because even if a certain area does not generate new events, the membrane potential state of the spiking neuron will remain for a period of time, allowing the network to remember the characteristics of the static area.

[0124] S303. Using a hierarchical point cloud encoding module to perform point-level feature extraction, local area feature extraction, and global feature extraction on the point cloud data;

[0125] Point-level feature extraction uses a multi-layer perceptron (MLP) to process each point independently; local region feature extraction captures the relationship between a point and its k nearest neighbors; global feature extraction obtains overall structural information through operations such as maximum pooling;

[0126] The hierarchical point cloud encoding module treats event data as a point cloud in the spatiotemporal domain, capturing both local and global features of the point cloud through multi-level abstraction. This module processes point cloud data in three stages: point-level feature extraction, local region feature extraction, and global feature extraction. This hierarchical design enables the network to fully understand the structure of the event point cloud, providing rich 3D information for human pose estimation.

[0127] S4. Dynamic Feature Enhancement and Cross-modal Fusion: After feature extraction, dynamic feature enhancement is performed using the dynamic feature enhancement module, and cross-modal fusion is performed using the cross-modal adaptive fusion module to optimize and integrate features from different modalities. This includes:

[0128] S401. Dynamic Feature Enhancement: The dynamic feature enhancement module uses channel attention and spatial attention mechanisms to adaptively enhance event features and spatial features based on the statistical information of input features to enhance key features;

[0129] The mathematical representation of the channel attention mechanism is as follows:

[0130] ;

[0131] The mathematical representation of the spatial attention mechanism is as follows:

[0132]

[0133] Among them, F is the input feature map, AvgPool is the global average pooling, MaxPool is the global maximum pooling, W1 and W2 are weight matrices, δ is the ReLU activation function, σ is the Sigmoid activation function, and [;] represents splicing in the channel dimension.

[0134] This dynamic feature enhancement mechanism can significantly increase the network's attention to key features and enhance representation capabilities.

[0135] S402. Cross-modal fusion: A cross-modal adaptive fusion module is used to perform cross-modal fusion of features. This module implements adaptive fusion through the following steps:

[0136] S4021. Modality-Specific Encoding: Spatial and event features are processed separately through dedicated encoders to enhance their respective representation capabilities.

[0137] S4022. Adaptive Weight Generation: Generate dynamic weights through global pooling and multi-layer perceptron to achieve dynamic weighting between modalities. The weight calculation formula is as follows:

[0138] ;

[0139] S4023. Weighted Fusion: Weights the features of different modalities according to the generated weights; expressed as follows:

[0140] ;

[0141] S4024. Cross-modal Attention Interaction: Deep Interaction between Modalities via Multi-Head Self-Attention Mechanism;

[0142] ;

[0143] in, F weighted Represents the feature fusion result after weighting. F s and F e are spatial features and event features respectively, w=[w1, w2] is the weight of the two modalities, Q, K, V are query, key, and value matrices, is the scaling factor.

[0144] The cross-modal adaptive fusion module is one of the core innovations of this invention, which is used to integrate features from different modalities. This adaptive fusion mechanism enables the network to dynamically adjust its attention to different modalities based on the characteristics of the input data, thereby adapting to various scenarios and motion patterns.

[0145] S5. Skeleton Constraint and Spatiotemporal Processing: After feature fusion, the skeleton constraint module and spatiotemporal joint processing module are used to enhance the network's understanding of human structure and modeling of spatiotemporal dependencies. Specific operations include the following:

[0146] S501. Skeleton constraint module:

[0147] The skeleton constraint module represents the human skeleton as a graph structure and learns the dependencies between joints through a graph convolutional network. The mathematical representation of the standard graph convolution is as follows:

[0148]

[0149] Where X' is the output feature, X is the input feature, W is the weight matrix, b is the bias, à = A + I is the adjacency matrix with self-connection added, and the degree matrix of à is D̃;

[0150] In this step, the spiking neural network is combined with graph convolution to design a spiking graph convolution layer. The spiking neuron mechanism is introduced into the standard graph convolution operation. The process is as follows:

[0151] First perform the standard graph convolution operation:

[0152] ;

[0153] The result Z is input to the spiking neuron:

[0154] ;

[0155] Where X is the input feature, W is the weight matrix, b is the bias, Ã = A + I is the adjacency matrix with self-connection added, D̃ is the degree matrix of Ã, and SpikeNeuron( ) is the spike activation function;

[0156] In order to enhance the rationality of human body structure, a bone length consistency loss function is designed:

[0157] ;

[0158] Here, bone refers to the line segment connecting two key points (joints) in the human skeleton model, bones refers to the set of all bones, that is, all the connecting line segments that constitute the complete human skeleton, length(bone) is the length of the bone, and std represents the standard deviation.

[0159] The skeleton constraint module also introduces a bone consistency loss, which encourages the consistency of bone length by calculating the within-batch standard deviation of the predicted bone length: this module significantly improves the robustness of the network in complex scenes by introducing prior knowledge of human anatomy, especially when dealing with partially occluded or blurred scenes.

[0160] S502. Joint spatiotemporal processing module: This module includes two key components: multi-scale temporal encoding and multi-head spatiotemporal attention, which are used to capture spatiotemporal dependencies in event data.

[0161] Multi-scale temporal coding uses convolution kernels of different sizes to process the time dimension and combines it with frequency domain feature extraction (implemented through FFT). That is, it uses a hybrid feature extraction method that combines a parallel convolution structure with frequency domain analysis. The mathematical expression is as follows:

[0162]

[0163] Among them, F is the input feature, F time is the time domain feature, F freq is the frequency domain feature, F out is the output feature, Conv k represents the convolution with kernel size k, [;] is the splicing in the channel dimension, W fusion is fused convolution, FFT is fast Fourier transform, LN is layer normalization, and g is a learnable gating parameter.

[0164] Multi-head spatiotemporal attention establishes long-distance dependencies between different spatiotemporal positions through the self-attention mechanism. The formula is as follows:

[0165]

[0166] in, F attn is the attention output, G spatial It is spatial gating, G temporal is time gating, and ⊙ is element-wise multiplication.

[0167] The above modules work together to enable the network to effectively capture the spatiotemporal dependencies in event data and improve its ability to understand complex action sequences.

[0168] S6. Keypoint Prediction and Training: A prediction head module is used to output the final coordinates of the keypoints of the human body. The prediction head module includes separate X-axis and Y-axis prediction branches, which predict the coordinates of the keypoints in two directions respectively. The prediction formula is as follows:

[0169]

[0170] Among them, x pred 、y pred is the predicted joint point coordinate, conf is the confidence, σ is the Sigmoid activation function, F global Represents global features, Project x 、 Project y 、 Project conf are the horizontal and vertical coordinates and the confidence predictor respectively, W and H are the width and height of the output feature map resolution;

[0171] S7. Using the preprocessed data from step S1, repeat steps S2 to S6 for training. The training method includes a three-level loss function and uses a surrogate gradient method for back propagation to achieve end-to-end training to obtain posture recognition results.

[0172] Specifically, the training of the multimodal neural network (UltraHybridSpikeNet) adopts a multi-task learning framework. The training method includes the collaborative optimization of three-level loss functions. The loss function is expressed as follows:

[0173] ;

[0174] in, L pose is the key point position loss, using the mean square error (MSE) to calculate the difference between the predicted key point and the true key point; λ bone represents the weight coefficient of the skeleton constraint loss (controlling the importance of bone length consistency loss in the total loss function), L bone is the skeleton constraint loss, which ensures that the predicted bone lengths meet physical constraints; λ temp represents the weight coefficient of temporal consistency loss (controlling the importance of smooth transition constraints between consecutive frames in the total loss function), L temp It is a temporal consistency loss that encourages smooth transitions between consecutive frames;

[0175] For the training of the SNN part, due to the non-differentiability of the impulse function, an alternative gradient method is used for backpropagation:

[0176] ;

[0177] This allows the entire network to be trained end-to-end via standard back-propagation.

[0178] See the flowchart Figure 2 , the structure diagram of the multimodal neural network framework is as follows Figure 3 shown.

[0179] By fully utilizing the spatiotemporal characteristics of event camera data, high-precision human pose estimation is achieved while maintaining low power consumption. This effectively addresses the challenges of existing methods, such as loss of temporal information, the "blind spot problem" in static areas, and the trade-off between computational efficiency and accuracy. This invention demonstrates significant advantages, particularly when dealing with fast motion and complex scenes, providing a viable solution for real-time human pose estimation in resource-constrained environments. The key technical points of this invention are as follows:

[0180] (1) A novel channel splitting strategy is proposed, which divides event data into spatial channels (x, y), event feature channels (T, P, E) and point cloud channels. Three dedicated feature extractors are designed to process different types of features respectively, maximizing the use of multi-dimensional information of events.

[0181] (2) A multi-gradient feature extraction module and a hierarchical point cloud encoding module were designed. Through multi-scale parallel convolution and hierarchical point cloud processing, the spatial multi-scale features and point cloud structural characteristics were effectively captured, significantly improving the spatial representation capability.

[0182] (3) A dedicated pulse feature processor was developed, which combined with advanced neuron models (adaptive LIF or Izhikevich) and time gating mechanisms to enhance the processing capability of event timing characteristics and effectively solve the "blind spot problem" in static areas.

[0183] (4) A cross-modal adaptive fusion module is introduced, which realizes the adaptive fusion and complex interaction of spatial features and event features through dynamic weight generation and multi-head attention mechanism, so that the network can dynamically adjust the processing strategy according to the characteristics of the input data.

[0184] (5) A skeleton constraint module based on pulse graph convolution was designed to integrate prior knowledge of human anatomy into the network. The rationality of the predicted posture was ensured through skeleton consistency loss, thereby improving the robustness of the model in complex scenes.

[0185] (6) A multi-scale temporal coding and multi-head spatiotemporal attention mechanism are proposed to enhance the network's ability to understand complex action sequences by capturing features at multiple time scales and establishing spatiotemporal dependencies.

[0186] (7) For the DHP19 dataset, an innovative five-channel input processing method (x, y, T, P, E) is proposed to effectively distinguish spatial and event features and provide the model with richer spatiotemporal information.

[0187] The advantages are: (1) Multi-dimensional information utilization: Existing methods such as EventPointPose and LiftMonoHPE mainly focus on spatial features or simple temporal features, while ignoring the multi-dimensional characteristics of event data. The present invention achieves comprehensive utilization of spatial, temporal, and structural information through a channel splitting strategy and a specially designed processor, significantly improving the accuracy of pose estimation.

[0188] (2) Static Region Representation Capability: Traditional event processing methods have a "blind spot problem" in static regions, where no events are generated in static regions and thus disappear in the representation. The dedicated pulse feature processor of the present invention effectively solves this problem by maintaining the membrane potential state of neurons.

[0189] (3) Robustness improvement: The skeleton constraint module of the present invention significantly improves the robustness of the model in partial occlusion, complex background and multi-person scenes by introducing prior knowledge of human anatomy.

[0190] (4) Dynamic Adaptability: The cross-modal adaptive fusion module of this invention dynamically adjusts the weights of different modalities based on the characteristics of the input data, enabling the network to adapt to various scenarios and motion patterns. This design excels in handling drastically changing speed and lighting conditions, and improves stability compared to fixed-weight approaches.

[0191] (5) Temporal consistency: Through multi-scale temporal coding and spatiotemporal attention mechanism, the present invention can better capture the temporal dependencies in continuous action sequences and reduce jitter and incoherence problems.

[0192] Through the innovative hybrid pulse-point cloud neural network architecture, the key problems in the existing event-driven human pose estimation method are solved, high accuracy and high robustness are achieved while maintaining low power consumption, providing a feasible solution for real-time human pose estimation in resource-constrained environments and has broad application prospects.

[0193] In summary, the present invention proposes a new neural network architecture, which aims to achieve efficient and accurate event-driven human posture estimation through innovative design. First, different from the existing methods that process event characteristics in a mixed manner, the present invention innovatively proposes a three-channel separation strategy, which divides event characteristics into spatial channels, event characteristic channels and point cloud channels, and designs special processors to extract features of different dimensions in a targeted manner, maximizing the use of multi-dimensional information of events. Secondly, the present invention adopts a multi-gradient feature extraction module to process spatial channels, effectively capturing spatial dependencies through multi-scale receptive fields; designs a dedicated pulse feature processor to process event characteristic channels, combines advanced neuron models and time gating mechanisms to efficiently process event timing characteristics, and solve the "blind spot problem" in static areas; processes point cloud representations through hierarchical point cloud encoders to capture the three-dimensional structural characteristics of events. Thirdly, the present invention develops a cross-modal adaptive fusion module, which realizes the intelligent integration of different modal features through dynamic weight generation and attention mechanism, so that the network can adaptively adjust the processing strategy according to the characteristics of the input data, and enhance the ability to understand complex scenes. Finally, the present invention designs a skeleton constraint module based on impulse graph convolution, incorporating prior knowledge of human anatomy into the network to ensure the biophysical plausibility of the predicted pose and improve robustness under occlusion and complex backgrounds. In summary, the present invention aims to provide a hybrid architecture that balances computational efficiency and accuracy, offering an efficient and feasible solution for real-time human pose estimation on edge devices.

[0194] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0195] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A human posture recognition method based on a multimodal neural network framework, characterized by: The specific steps include: S1. Data preprocessing: Event data is collected and represented using a five-channel event representation method: E' = (x, y, T, P, E). Where x and y are pixel positions, T is the average timestamp, P is the cumulative positive, and E is the event count; S2. Channel Splitting and Trimodal Separation: The five-channel event data preprocessed in step S1 is divided into three channels: a spatial channel containing normalized spatial coordinates (x, y); an event feature channel containing temporal and polarity features (T, P, E); and a point cloud channel: the complete five-channel data is treated as point cloud data, preserving the original three-dimensional characteristics of the event. S3. Multimodal feature extraction: After completing channel splitting, feature extraction is performed on the data of the spatial channel, event characteristic channel, and point cloud channel respectively; S4. Dynamic Feature Enhancement and Cross-modal Fusion: After feature extraction, dynamic feature enhancement and cross-modal fusion are used to optimize and integrate features from different modalities. This includes: S401. Dynamic Feature Enhancement: The dynamic feature enhancement module uses channel attention and spatial attention mechanisms to adaptively enhance event features and spatial features based on the statistical information of input features to enhance key features; S402. Cross-modal fusion: Using a cross-modal adaptive fusion module to perform cross-modal fusion of features; S5. Skeleton Constraint and Spatiotemporal Processing: After feature fusion, the skeleton constraint module and spatiotemporal joint processing module are used to enhance the network's understanding of human structure and spatiotemporal dependencies. The skeleton constraint module combines spiking neural networks with graph convolution, introducing a spiking neuron mechanism into the standard graph convolution operation. The process is as follows: First perform the standard graph convolution operation: ; Input the result Z into the pulse neuron: ; Where X is the input feature, W is the weight matrix, b is the bias, Ã=A+I is the adjacency matrix with self-connection added, D̃ is the degree matrix of Ã, and SpikeNeuron() is the spike activation function; In order to enhance the rationality of human body structure, a bone length consistency loss function is designed: ； Among them, bone is the line segment connecting two key point joints in the human skeleton model, bones is all the connecting line segments that constitute the complete human skeleton, length(bone) is the length of the bone, and std represents the standard deviation; S6. Keypoint prediction and training: Use the prediction head module to output the final coordinates of the human body keypoints; S7. Using the preprocessed data from step S1, repeat steps S2 to S6 for training. The training method includes a three-level loss function and uses the substituted gradient method for backpropagation to achieve end-to-end training to obtain the posture recognition results. The loss function is expressed as follows: ； in, L pose is the keypoint location loss, which uses the mean squared error to calculate the difference between the predicted keypoint and the true keypoint; λ bone represents the weight coefficient of the skeleton constraint loss, L bone is the skeleton constraint loss, which ensures that the predicted bone lengths meet physical constraints; λ temp represents the weight coefficient of temporal consistency loss, L temp is a temporal consistency loss that encourages smooth transitions between consecutive frames.

2. The method for human posture recognition based on a multimodal neural network framework according to claim 1, wherein: The step S3 specifically includes: S301. Processing spatial channel data using a multi-gradient feature extraction module; S302. Processing event characteristic channel data using a pulse feature processor; S303. Use a hierarchical point cloud encoding module to perform point-level feature extraction, local area feature extraction, and global feature extraction on the point cloud data.

3. The method for human posture recognition based on a multimodal neural network framework according to claim 2, wherein: The multi-gradient feature extraction module in step S301 includes four parallel branches, namely, a small-scale feature extraction convolution kernel, a medium-scale feature extraction convolution kernel, a large-scale feature extraction convolution kernel, and an ultra-large-scale feature extraction convolution kernel, and the convolution kernel sizes are 1×1, 3×3, 5×5, and 7×7, respectively; the outputs of each branch are spliced and fused after batch normalization and activation function processing.

4. The method for human posture recognition based on a multimodal neural network framework according to claim 2, wherein: The point-level feature extraction in step S303 processes each point independently through a multi-layer perceptron; the local area feature extraction captures the relationship between the point and its k nearest neighbors; and the global feature extraction obtains the overall structural information through a maximum pooling operation.

5. A method for human posture recognition based on a multimodal neural network framework according to any one of claims 2 to 4, characterized in that: The pulse feature processor in step S302 includes an adaptive LIF neuron or an Izhikevich neuron; the dynamic equation of the adaptive LIF neuron is as follows: Membrane potential update: Threshold Update: Pulse Generation: Where spike is the action potential of the neuron, v is the membrane potential, θ is the adaptive threshold, and τ mem is the membrane potential time constant, τ thresh is the threshold time constant, θ rest is the resting threshold, I is the input current, and v t represents the membrane potential at time t, θ t represents the adaptive threshold at time t, I t represents the input current at time t, and Heaviside is a step function; The dynamic equation of the Izhikevich neuron is: Pulse trigger condition: if v ≥ threshold, then spike = 1, otherwise spike = 0; reset after pulse: if spike = 1, then v = c, u = u + d; Where v is the membrane potential, u is the membrane potential recovery variable, I is the input current, and a, b, c, and d are parameters that control neuronal behavior.

6. The method for human posture recognition based on a multimodal neural network framework according to claim 5, characterized in that: The pulse feature processor in step S302 also includes a time gating unit and a bidirectional LSTM layer to further extract timing features and long-term dependencies; the specific operations are as follows: the event feature channel (T, P, E) data is convolutionally preprocessed through a convolution pulse encoder; the convolved features are processed using an adaptive LIF neuron model to enhance the processing capability of timing information; the adaptively processed features are subjected to timing modeling through a bidirectional LSTM layer to capture the forward and backward dependencies of the time series; the processing capability of time information is further enhanced through the time gating unit to control the flow of information.

7. The method for human posture recognition based on a multimodal neural network framework according to claim 1, wherein: The mathematical representation of the channel attention mechanism in step S401 is as follows: ； The mathematical representation of the spatial attention mechanism is as follows: Where F is the input feature map, AvgPool is the global average pooling, MaxPool is the global maximum pooling, W1 and W2 are weight matrices, δ is the ReLU activation function, σ is the Sigmoid activation function, and [;] represents splicing in the channel dimension; In step S402, the cross-modal adaptive fusion module implements adaptive fusion through the following steps: S4021. Modality-Specific Encoding: Spatial and event features are processed separately through dedicated encoders to enhance their respective representation capabilities. S4022. Adaptive Weight Generation: Generate dynamic weights through global pooling and multi-layer perceptron to achieve dynamic weighting between modalities. The weight calculation formula is as follows: ； S4023. Weighted Fusion: Weight the features of different modalities according to the generated weights; ； S4024. Cross-modal Attention Interaction: Deep Interaction between Modalities via Multi-Head Self-Attention Mechanism; ； in, F weighted Represents the feature fusion result after weighting. F s and F e are spatial features and event features respectively, w=[w1, w2] is the weight of the two modalities, Q, K, V are query, key, and value matrices, is the scaling factor.

8. The method for human posture recognition based on a multimodal neural network framework according to claim 1, wherein: In step S5, a spatiotemporal joint processing module is used to capture the spatiotemporal dependencies in the event data; the spatiotemporal joint processing module includes multi-scale temporal coding and multi-head spatiotemporal attention; The multi-scale temporal coding adopts a hybrid feature extraction method that combines a parallel convolution structure with frequency domain analysis. The parallel convolution structure includes convolution kernels of different sizes for processing the time dimension. The frequency domain analysis uses fast Fourier transform to extract frequency domain features. The expression is as follows: Among them, F is the input feature, F time is the time domain feature, F freq is the frequency domain feature, F out is the output feature, Conv k represents the convolution with kernel size k, [;] is the splicing in the channel dimension, W fusion is fused convolution, FFT is fast Fourier transform, LN is layer normalization, and g is a learnable gating parameter; Multi-head spatiotemporal attention establishes long-distance dependencies between different spatiotemporal positions through the self-attention mechanism. The formula is as follows: in, F attn is the attention output, G spatial It is spatial gating, G temporal is time gating, and ⊙ is element-wise multiplication.

9. The method for human posture recognition based on a multimodal neural network framework according to claim 1, wherein: The prediction formula of the prediction head module in step S6 is as follows: Among them, x pred 、y pred is the predicted joint point coordinate, conf is the confidence, σ is the Sigmoid activation function, F global Represents global features, Project x 、 Project y 、 Project conf are the coordinate and confidence predictors respectively, W and H are the width and height of the output feature map resolution.

Images